National platforms #57

ocefpaf · 2024-02-15T12:49:09Z

Ported from #56:

HF Radar Stations # This is a hardcoded number in the notebook (165), is that still valid @MathewBiddle?
National Platforms # Lots of services and should be into a separate module
NGDAC Glider Days # Implemented but must change to glider-data-days
rename run_metrics.ipynb to IOOS_BTN.ipynb when all PRs are approved.

CO-OPS

This code returns 381 stations but searvey reports a lower number (366) of active stations and higher total if we add the discontinued ones (446):

from searvey import coops
df = coops.coops_stations()
len(df.loc[df["status"] == "active"])
366
len(df.loc[df["status"] == "discontinued"])
80

Not sure which data source is better to answer that question. Searvey does have this warning but I'm not sure if that means the API they are hitting is outdated in terms of usability or data updates.

NERRS

The code returns 93 but there is a comment in the notebook that it should be around 140

PS: The current refactor is used in this example.

ocefpaf · 2024-04-03T19:48:24Z

@MathewBiddle we need to be a bit careful with this one b/c I had two difficult rebasing to catch up with #56 and #54. We should check if we still have all the metrics functions expected and/or if we don't have duplicates.

Once we merge this I'll send a final PR making it a package, publish, and then folks can run any metric they want by just installing this and calling the proper function.

MathewBiddle · 2024-04-08T14:03:29Z

gliderpy is throwing a user warning

C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\gliderpy\fetchers.py:47: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
  df.index = pd.to_datetime(df.index)

ocefpaf · 2024-04-08T18:01:47Z

gliderpy is throwing a user warning

I have plans to investigate that later but it is not gliderpy it is some bad dates in the data that cannot be parsed with vanilla pandas. In theory, it is OK. However, as we are closely responsible for that date, we should investigate. I'll let you know what I find soon but I need to read every time variable data point to find which one is triggering that... So I'm not doing that today 😬

MathewBiddle · 2024-04-08T19:40:29Z

looks like I'm getting a timeout from, https://opendap.co-ops.nos.noaa.gov/stations/stationsXML.jsp.

df2 = ioos_metrics.ioos_metrics.update_metrics()
C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\gliderpy\fetchers.py:47: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
  df.index = pd.to_datetime(df.index)
C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\gliderpy\fetchers.py:47: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
  df.index = pd.to_datetime(df.index)
C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\gliderpy\fetchers.py:47: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
  df.index = pd.to_datetime(df.index)
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\urllib3\connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\urllib3\connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 1423, in getresponse
    response.begin()
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\socket.py", line 707, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\ssl.py", line 1252, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\ssl.py", line 1104, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\urllib3\connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\urllib3\util\retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\urllib3\util\util.py", line 39, in reraise
    raise value
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\urllib3\connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\urllib3\connectionpool.py", line 539, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\urllib3\connectionpool.py", line 370, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='opendap.co-ops.nos.noaa.gov', port=443): Read timed out. (read timeout=10)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\externals\loky\process_executor.py", line 463, in _process_worker
    r = call_item()
        ^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\externals\loky\process_executor.py", line 291, in __call__
    return self.fn(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 589, in __call__
    return [func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\national_platforms.py", line 51, in get_coops
    xml = requests.get(
          ^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\requests\adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='opendap.co-ops.nos.noaa.gov', port=443): Read timed out. (read timeout=10)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\IPython\core\interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-da9d358039b7>", line 1, in <module>
    df2 = ioos_metrics.ioos_metrics.update_metrics()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 577, in update_metrics
    national_platforms = sum(
                         ^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 1595, in _get_outputs
    yield from self._retrieve()
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 1699, in _retrieve
    self._raise_error_fast()
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 1734, in _raise_error_fast
    error_job.get_result(self.timeout)
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 736, in get_result
    return self._return_or_raise()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 754, in _return_or_raise
    raise self._result
requests.exceptions.ReadTimeout: None: None

Url should be https://opendap.co-ops.nos.noaa.gov/stations/stationsXML.jsp (no clue what is different?)

ocefpaf · 2024-04-08T19:52:25Z

Url should be https://opendap.co-ops.nos.noaa.gov/stations/stationsXML.jsp (no clue what is different?)

If you try to navigate there you should get a 504. Let's see if the service is restored later today/tomorrow.

MathewBiddle · 2024-04-08T20:37:27Z

FYI, these tests have been running >13min...

ocefpaf · 2024-04-08T20:52:13Z

FYI, these tests have been running >13min...

The "slow" glider test takes that long. I'm waiting for Kathy's answer to figure out what to do with it.

ocefpaf · 2024-04-09T16:26:17Z

@MathewBiddle I made a few modifications to this PR after our slack conversation with Kathy:

The default for the glider metric in update_metrics is the fast method now, no more waiting for >30 min!
Both fast and slow glider methods are present but only slow returns profile count.
The difference between the gliders methods varies depending on the query parameters but the largest difference I computed was 1.3%, that is less than before b/c I had a mistake in the end_date and I was comparing a full query vs a constrained one.
We should not get any warning when parsing dates now. All dates are validated against ERDDAP's format and, if they aren't, the metric will fail and record the dataset_id in the logs.

MathewBiddle · 2024-04-16T20:24:26Z

I am trying to look through metric.log but there are a lot of pdfminer logs written. Below is the error I'm getting right now:

df2 = ioos_metrics.ioos_metrics.update_metrics()
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\urllib\request.py", line 1344, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 1331, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 1377, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 1326, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 1085, in _send_output
    self.send(msg)
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 1029, in send
    self.connect()
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 1465, in connect
    super().connect()
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\http\client.py", line 995, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\socket.py", line 852, in create_connection
    raise exceptions[0]
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\socket.py", line 837, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\externals\loky\process_executor.py", line 463, in _process_worker
    r = call_item()
        ^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\externals\loky\process_executor.py", line 291, in __call__
    return self.fn(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 589, in __call__
    return [func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 323, in regional_platforms
    df = pd.read_json(url)
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\pandas\io\json\_json.py", line 791, in read_json
    json_reader = JsonReader(
                  ^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\pandas\io\json\_json.py", line 904, in __init__
    data = self._get_data_from_filepath(filepath_or_buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\pandas\io\json\_json.py", line 944, in _get_data_from_filepath
    self.handles = get_handle(
                   ^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\pandas\io\common.py", line 728, in get_handle
    ioargs = _get_filepath_or_buffer(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\pandas\io\common.py", line 384, in _get_filepath_or_buffer
    with urlopen(req_info) as req:
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\pandas\io\common.py", line 289, in urlopen
    return urllib.request.urlopen(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\urllib\request.py", line 215, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\urllib\request.py", line 515, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\urllib\request.py", line 532, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\urllib\request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\urllib\request.py", line 1392, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\urllib\request.py", line 1347, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [WinError 10061] No connection could be made because the target machine actively refused it>
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\IPython\core\interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-da9d358039b7>", line 1, in <module>
    df2 = ioos_metrics.ioos_metrics.update_metrics()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 593, in update_metrics
    columns = dict(zip(functions.keys(), values, strict=False))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 1595, in _get_outputs
    yield from self._retrieve()
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 1699, in _retrieve
    self._raise_error_fast()
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 1734, in _raise_error_fast
    error_job.get_result(self.timeout)
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 736, in get_result
    return self._return_or_raise()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 754, in _return_or_raise
    raise self._result
urllib.error.URLError: <urlopen error [WinError 10061] No connection could be made because the target machine actively refused it>

ocefpaf · 2024-04-17T06:56:04Z

It is our usual ERDDAP server down ;-p

https://erddap.ioos.us/erddap/tabledap/

I'll improve the logging system to make it easier to stop network problems.

ocefpaf · 2024-04-17T13:08:23Z

Both the our logs and the CI logs are easier to parse now. This is the CI logs:

tests/test_metrics.py:67: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ioos_metrics/ioos_metrics.py:323: in regional_platforms
    df = pd.read_json(url)
../../../micromamba/envs/TEST/lib/python3.12/site-packages/pandas/io/json/_json.py:791: in read_json
    json_reader = JsonReader(
../../../micromamba/envs/TEST/lib/python3.12/site-packages/pandas/io/json/_json.py:904: in __init__
    data = self._get_data_from_filepath(filepath_or_buffer)
../../../micromamba/envs/TEST/lib/python3.12/site-packages/pandas/io/json/_json.py:944: in _get_data_from_filepath
    self.handles = get_handle(
../../../micromamba/envs/TEST/lib/python3.12/site-packages/pandas/io/common.py:728: in get_handle
    ioargs = _get_filepath_or_buffer(
../../../micromamba/envs/TEST/lib/python3.12/site-packages/pandas/io/common.py:384: in _get_filepath_or_buffer
    with urlopen(req_info) as req:
../../../micromamba/envs/TEST/lib/python3.12/site-packages/pandas/io/common.py:289: in urlopen
    return urllib.request.urlopen(*args, **kwargs)
../../../micromamba/envs/TEST/lib/python3.12/urllib/request.py:215: in urlopen
    return opener.open(url, data, timeout)
../../../micromamba/envs/TEST/lib/python3.12/urllib/request.py:515: in open
    response = self._open(req, data)
../../../micromamba/envs/TEST/lib/python3.12/urllib/request.py:532: in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
../../../micromamba/envs/TEST/lib/python3.12/urllib/request.py:492: in _call_chain
    result = func(*args)
../../../micromamba/envs/TEST/lib/python3.12/urllib/request.py:1392: in https_open
    return self.do_open(http.client.HTTPSConnection, req,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPSHandler object at 0x7f690b211bb0>
http_class = <class 'http.client.HTTPSConnection'>
req = <urllib.request.Request object at 0x7f69010865d0>
http_conn_args = {'context': <ssl.SSLContext object at 0x7f69080c3b50>}
host = 'erddap.ioos.us'
h = <http.client.HTTPSConnection object at 0x7f6901086630>
headers = {'Connection': 'close', 'Host': 'erddap.ioos.us', 'User-Agent': 'Python-urllib/3.12'}

Note that network issues here are expected and I hope to run this as a cron job to catch them early. The code per se is ready for review.

MathewBiddle · 2024-04-17T14:09:26Z

After rebooting ERDDAP (ioos/erddap-gold-standard#69), this is running successfully now.

ioos_metrics/ioos_metrics.py

MathewBiddle · 2024-04-30T18:11:23Z

Testing this PR and I run into an issue with missing atn.

df2 = ioos_metrics.ioos_metrics.update_metrics()
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\externals\loky\process_executor.py", line 463, in _process_worker
    r = call_item()
        ^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\externals\loky\process_executor.py", line 291, in __call__
    return self.fn(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 589, in __call__
    return [func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 335, in atn_deployments
    return atn
           ^^^
UnboundLocalError: cannot access local variable 'atn' where it is not associated with a value
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\IPython\core\interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-da9d358039b7>", line 1, in <module>
    df2 = ioos_metrics.ioos_metrics.update_metrics()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 575, in update_metrics
    columns = dict(zip(functions.keys(), values, strict=False))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 1595, in _get_outputs
    yield from self._retrieve()
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 1699, in _retrieve
    self._raise_error_fast()
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 1734, in _raise_error_fast
    error_job.get_result(self.timeout)
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 736, in get_result
    return self._return_or_raise()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\joblib\parallel.py", line 754, in _return_or_raise
    raise self._result
UnboundLocalError: cannot access local variable 'atn' where it is not associated with a value

ioos_metrics/ioos_metrics.py

MathewBiddle · 2024-04-30T18:28:00Z

This looks good now. I'm merging so we can move forward. However, ATN website needs to be updated and then the full update_metrics can run.

pdf miner logs are still unbearable.

MathewBiddle

I approve this knowing that ATN deployments will fail at the moment.

ocefpaf · 2024-04-30T20:17:53Z

I approve this knowing that ATN deployments will fail at the moment.

I'll open issue for cleaner logs and fix the ATN URL.

ocefpaf force-pushed the national_platforms branch from 760ca0d to dafe187 Compare February 15, 2024 14:16

ocefpaf mentioned this pull request Feb 15, 2024

Parallel #56

Merged

4 tasks

ocefpaf force-pushed the national_platforms branch from 3575f61 to ead61f8 Compare February 15, 2024 15:35

ocefpaf force-pushed the national_platforms branch 4 times, most recently from 0e059b0 to 81bf01c Compare April 3, 2024 19:14

ocefpaf marked this pull request as ready for review April 3, 2024 19:44

ocefpaf force-pushed the national_platforms branch from 81bf01c to d924b1f Compare April 8, 2024 18:02

ocefpaf force-pushed the national_platforms branch from 5d91062 to d53ae59 Compare April 8, 2024 20:22

ocefpaf added 6 commits April 17, 2024 09:03

add hf radar, national_platforms and slow but accurate glider metric

1fe589a

pull latest gliderpy

2611749

add date selection and full metrics in the slow version for gliders

a799a3a

use fast version in the metrics

80edd0c

don't run the slow glider test

917b077

re-lock

d0182a5

ocefpaf force-pushed the national_platforms branch from a4a1705 to d0182a5 Compare April 17, 2024 07:06

ocefpaf mentioned this pull request Apr 17, 2024

Pkg metrics #64

Merged

2 tasks

MathewBiddle reviewed Apr 17, 2024

View reviewed changes

ioos_metrics/ioos_metrics.py Outdated Show resolved Hide resolved

ocefpaf force-pushed the national_platforms branch from 08ddbc5 to 03ac038 Compare April 17, 2024 14:11

better error reporting

32588ce

ocefpaf force-pushed the national_platforms branch from 03ac038 to 32588ce Compare April 18, 2024 15:30

factor out national platforms call

9639aea

MathewBiddle reviewed Apr 30, 2024

View reviewed changes

ioos_metrics/ioos_metrics.py Show resolved Hide resolved

MathewBiddle approved these changes Apr 30, 2024

View reviewed changes

MathewBiddle merged commit 83b878b into ioos:main Apr 30, 2024
2 checks passed

ocefpaf deleted the national_platforms branch April 30, 2024 20:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

National platforms #57

National platforms #57

ocefpaf commented Feb 15, 2024 •

edited

Loading

ocefpaf commented Apr 3, 2024

MathewBiddle commented Apr 8, 2024

ocefpaf commented Apr 8, 2024

MathewBiddle commented Apr 8, 2024

ocefpaf commented Apr 8, 2024

MathewBiddle commented Apr 8, 2024

ocefpaf commented Apr 8, 2024

ocefpaf commented Apr 9, 2024

MathewBiddle commented Apr 16, 2024

ocefpaf commented Apr 17, 2024

ocefpaf commented Apr 17, 2024

MathewBiddle commented Apr 17, 2024

MathewBiddle commented Apr 30, 2024

MathewBiddle commented Apr 30, 2024

MathewBiddle left a comment

ocefpaf commented Apr 30, 2024

National platforms #57

National platforms #57

Conversation

ocefpaf commented Feb 15, 2024 • edited Loading

CO-OPS

NERRS

ocefpaf commented Apr 3, 2024

MathewBiddle commented Apr 8, 2024

ocefpaf commented Apr 8, 2024

MathewBiddle commented Apr 8, 2024

ocefpaf commented Apr 8, 2024

MathewBiddle commented Apr 8, 2024

ocefpaf commented Apr 8, 2024

ocefpaf commented Apr 9, 2024

MathewBiddle commented Apr 16, 2024

ocefpaf commented Apr 17, 2024

ocefpaf commented Apr 17, 2024

MathewBiddle commented Apr 17, 2024

MathewBiddle commented Apr 30, 2024

MathewBiddle commented Apr 30, 2024

MathewBiddle left a comment

Choose a reason for hiding this comment

ocefpaf commented Apr 30, 2024

ocefpaf commented Feb 15, 2024 •

edited

Loading